feat(viewer,webview): embed QtMultimedia in AnthiasWebview, eliminate two-process DRM contention + Pi 4 drops#2905
Open
vpetersson wants to merge 10 commits into
Open
feat(viewer,webview): embed QtMultimedia in AnthiasWebview, eliminate two-process DRM contention + Pi 4 drops#2905vpetersson wants to merge 10 commits into
vpetersson wants to merge 10 commits into
Conversation
…cess Routes Qt6-board video playback through ``mpv_render_context`` inside the AnthiasWebview process so a single Qt process holds DRM master / the Wayland surface for both web pages, images, and video. Fixes the two-process framebuffer contention that produced 600-2800 vo drops per 60 s clip on Pi 4 under linuxfb (issue #2904). - ``src/anthias_webview/`` gains ``VideoView`` (``QOpenGLWidget`` + libmpv ``mpv_render_context``) and two new D-Bus slots on ``MainWindow`` (``playVideo`` / ``stopVideo``) plus a ``videoEnded`` signal. ``View`` toggles VideoView visibility alongside the existing dual ``QWebEngineView`` / image surface. - ``src/anthias_viewer/media_player.py``: ``MPVMediaPlayer.play`` / ``stop`` switch from ``subprocess.Popen([mpv, ...])`` to ``browser_bus.playVideo(uri, options)`` over the same pydbus proxy the viewer already uses for ``loadPage`` / ``loadImage``. Options dict carries ``hwdec`` / ``audio-device`` / ``video-sync`` / ``vd-lavc-threads`` / ``video-rotate`` — the per-board, per-codec dispatch and the ALSA / rotation rules from PR #2885 are preserved verbatim. ``--vo=...`` and ``--drm-mode=...`` are gone: libmpv-render paints into Qt's GL FBO and Qt's eglfs KMS config (Pi 4) pins the framebuffer mode now. - ``docker/Dockerfile.viewer.j2``: ``libmpv-dev`` + ``pkg-config`` in the builder, ``libmpv2`` at runtime (dropping the unused ``mpv`` CLI — ``ffprobe`` ships in the ``ffmpeg`` package). Pi 4 switches ``QT_QPA_PLATFORM`` from ``linuxfb`` → ``eglfs`` so Qt has the GL context libmpv-render needs; eglfs KMS config pins ``HDMI1``/ ``HDMI2`` to 1920x1080 and ``QT_SCALE_FACTOR=1`` short-circuits the auto-detect that would otherwise pick 2x from the 4K connector EDID. - ``tests/test_media_player.py``: argv assertions become options-dict assertions; ``subprocess.Popen`` mock becomes a ``_browser_bus`` mock. ``is_playing()`` flips to a local flag (the asset_loop has zero callers — only tests). - ``src/anthias_webview/tests/``: new QtTest unit suite for ``VideoView`` (libmpv handle lifecycle, option round-trip via ``mpv_get_property_string``, ``--key`` → ``key`` normalisation, stop-without-play idempotence, video-rotate value passthrough). ``bin/test_webview_cpp.sh`` builds and runs them under ``QT_QPA_PLATFORM=offscreen``. Not wired into CI in this PR.
pydbus refuses to coerce a plain Python ``str`` value into a
``GLib.Variant`` when the slot is declared with ``QVariantMap``
(D-Bus ``a{sv}``), surfacing at runtime as
``MPVMediaPlayer.play failed: argument value: Expected GLib.Variant,
but got str`` on every video play. Wrap each option value via
``_marshal_dbus_options`` so the call site hands pydbus a
``dict[str, GLib.Variant]``. Test fixture monkeypatches the helper
to identity so existing options-dict assertions keep working;
a regression test verifies the wrap actually produces ``GLib.Variant``
instances (caught the same way the live error surfaced on the
x86 testbed deploy).
Adds hard-data instrumentation so we can compare the libmpv-embedded path against PR #2885's 600-2800 vo drops/60 s Pi 4 baseline rather than inferring "no drops" from low load averages. VideoView opens ``/data/.anthias/mpv-stats.log`` (bind-mounted from the host so ``docker exec cat`` reaches it) and writes: - ``INIT`` at startup (libmpv client API version) - ``LOADFILE`` on every ``play()`` call — the requested option dict (hwdec, audio-device, video-sync, vd-lavc-threads, video-rotate) - ``FILE_LOADED`` after mpv's decoder probe — ``hwdec-current`` is the actual decoder mpv engaged (catches silent SW fallback when the requested hwdec didn't whitelist), plus video-codec / w / h / container-fps for the asset - ``SAMPLE`` every 1 s during playback (time-pos, frame-drop-count) - ``END_FILE`` on clip completion — final drop count + elapsed_ms - ``STOP`` when MPVMediaPlayer.stop() interrupts mid-play The 8-case QtTest suite still passes (mpv property round-trip, option normalisation, stop idempotence, video-rotate passthrough). The warnings about "/data/.anthias" being missing fire only on the dev host — production has the bind mount.
…tats file Diagnostic for HEVC drm-copy silently SW-falling-back under the render API on Pi 4. Subprocess mpv with --vo=null --hwdec=drm-copy engages the rpi-hevc-dec block via /dev/video19 + /dev/media0 fine, but the embedded libmpv path reports hwdec-current= empty for every HEVC clip while the same 1080p60 H.264 + v4l2m2m-copy engages cleanly. Verbose log + persisting MPV_EVENT_LOG_MESSAGE rows to mpv-stats.log lets us see exactly which hwdec init step is failing without depending on AnthiasWebview's stderr (which sh.Command on the Python side captures into an unreachable bytes buffer).
QOpenGLWidget renders into an offscreen FBO that Qt's compositor then blits into the window — two copies per frame on top of libmpv-render's GL upload. On Pi 4 V3D 6.0 that pushed 1080p60 H.264 to 2973 drops/60 s even with HW decode confirmed engaged (``[vd] Using hardware decoding (v4l2m2m-copy)``); the bottleneck was VO throughput, not the decoder. ``QOpenGLWindow`` with ``NoPartialUpdate`` swaps the native window's default framebuffer directly — same path mpv's own Qt example uses. VideoView changes from ``QOpenGLWidget : QWidget`` to ``QOpenGLWindow : QPaintDeviceWindow`` (QWindow tree, not widget tree). ``View`` wraps it with ``QWidget::createWindowContainer`` so MainWindow's existing widget-tree layout (dual QWebEngineView pair + image canvas + video) stays intact. Visibility / geometry toggle on the container; the GL render context lives on the inner QWindow and persists across show/hide so repeated plays don't re-initialise ``mpv_render_context``. ``QT += opengl`` replaces ``openglwidgets`` in both ``AnthiasWebview.pro`` and ``tests/tests.pro``. 8 QtTest cases still pass (mpv handle lifecycle, option round-trip, key normalisation, stop idempotence, video-rotate passthrough), and the offscreen-platform "QOpenGLWidget not supported" warning that fired on the dev host is gone — QOpenGLWindow doesn't have that restriction.
…ndow" This reverts commit bd558c9.
The libmpv-embedded path engaged HW decode correctly on all 4 Qt6 boards but didn't move Pi 4 frame drops below the subprocess-mpv baseline (562–2973 drops/60 s, same range as PR #2885). Real-device verbose mpv logging confirmed the decoders engaged; the bottleneck was V3D 6.0 fillrate through libmpv-render → QOpenGLWidget FBO → Qt-compositor → eglfs swap. Skipping the FBO indirection by porting to QOpenGLWindow crashed under eglfs's single-native-window-per- process limit (reverted at f057198). QtMultimedia + gstreamer is the next try: - ``VideoView`` rewritten around QMediaPlayer + QVideoWidget + QAudioOutput. QVideoWidget paints inside MainWindow's existing eglfs native window, so we don't trip the single-window restriction. Stats logger keeps the same /data/.anthias/mpv-stats.log schema (INIT / LOADFILE / PLAYING / SAMPLE / END_FILE) with a drop estimate computed from container_fps × elapsed − frames- delivered (QVideoSink::videoFrameChanged counter). - ``QT_MEDIA_BACKEND=gstreamer`` is set in the viewer Dockerfile so Qt picks the gstreamer backend over its ffmpeg one — the rpi ``v4l2slh264dec`` / ``v4l2slh265dec`` elements (in rpt3 ``gstreamer1.0-plugins-bad``, confirmed via ``dpkg-deb -c`` on the .deb) route directly to QVideoSink. - ``docker/_rpt1-ffmpeg-pin.j2`` extends the rpi-archive pin to ``gstreamer1.0-*`` + ``libgstreamer*`` so plugins-bad wins from rpt3 over stock Debian (priority bump 100 → 1001). - ``viewer_extra_apt_dependencies`` swaps libmpv2 for the gstreamer1.0-{alsa,libav,plugins-{base,good,bad,ugly}} + libqt6multimedia6 + libqt6multimediawidgets6 + qt6-multimedia-dev set. - ``MPVMediaPlayer`` Python options shrink to audio-device + video-rotate (Pi 4 only). Removed: hwdec, video-sync, vd-lavc-threads, the _PI_HWDEC_BY_CODEC table, _probe_video_codec, _pi_hwdec_for_uri. Codec dispatch is now gstreamer's job. - Python tests drop the ~12 per-codec / ffprobe-dispatch tests; C++ tests drop the mpv_get_property_string round-trip in favour of QMediaPlayer construction + option-passthrough assertions. Codec-gate symmetry test replaced with "gate codecs ⊆ {h264, hevc}" — broader, catches a relaxation of the upload gate at the same time. Pi 4 perf gain is unverified — Pi 4 testbed went offline mid-session (along with Pi 5 and Rock Pi 4). Real-device validation pending.
… env End-to-end test on x86 caught two bugs: 1. ``player->setSource(QUrl(uri))`` parsed local paths (``/data/anthias_assets/...mp4``) as scheme-less relative URLs. QMediaPlayer's ffmpeg backend silently refuses to load them — no error fired, position stayed at 0 for the whole 12-second asset_loop wait. Use ``QUrl::fromLocalFile`` when the URI starts with ``/``; absolute URLs (``http://``, ``file://``, ``rtsp://``) round-trip through ``QUrl(uri)`` as before. 2. Qt 6.8 dropped the gstreamer multimedia backend upstream (6.5+). Debian Trixie ships only ``libffmpegmediaplugin.so`` in ``/usr/lib/.../qt6/plugins/multimedia/``. ``QT_MEDIA_BACKEND= gstreamer`` silently fell back to ffmpeg, which means our path is fundamentally similar to the libmpv-via-render-API attempt (ffmpeg decoder → QtMultimedia GL upload). The plan's "zero-copy DMA-BUF through gstreamer" claim doesn't hold on this Qt version. Pin ``QT_MEDIA_BACKEND=ffmpeg`` explicitly so a future Qt that re-introduces the gstreamer backend doesn't silently switch us. Keep the ``gstreamer1.0-*`` apt set because the +rpt1 libavcodec the ffmpeg backend dlopens IS configured with v4l2_request / v4l2m2m hwaccels, and ``gst-launch-1.0`` / ``gst-inspect-1.0`` stay useful as hardware diagnostics. Pi 4 perf gain still unverified — same architectural ceiling applies. The PR continues as architectural cleanup (single Qt process, no mpv subprocess, smaller option dict) while we measure.
The earlier commit added the full gstreamer1.0-* plugin set under the assumption QtMultimedia would route through ``v4l2slh*dec`` elements. Qt 6.8 dropped that backend upstream — only ``libffmpegmediaplugin.so`` ships in ``/usr/lib/.../qt6/plugins/multimedia/``. Decode actually goes through libavcodec directly (the +rpt1 build still carries ``--enable-v4l2-request`` / ``--enable-v4l2-m2m``, so HW decode still reaches the rpi-hevc-dec + bcm2835-codec hardware), and the gstreamer packages were dead weight: ~400-500 MB of image bloat that pushed the Pi 4 viewer image past the SD card's free space during ``docker load`` (disk hit 0 % before the load completed). Drop the gstreamer set + the ``QT_MEDIA_BACKEND=ffmpeg`` env (default selection arrives at ffmpeg anyway, no need to pin). Also trim ``_rpt1-ffmpeg-pin.j2``'s pin string back to the ffmpeg / libav* family.
view_video in src/anthias_viewer/__init__.py:495 calls
``view_image('null')`` AFTER ``media_player.play()`` to clear any
leftover image / webpage background while the just-started video
holds the foreground. My loadImage handler was unconditionally
calling ``hideVideoSurface()`` (→ ``videoView->stop()``) on every
loadImage, including the ``'null'`` sentinel, which stopped the
video 66 ms after PLAYING fired.
On Pi 4 this was mostly invisible because the bcm2835-codec
decoder is past its critical init phase at 66 ms; the player
recovered. On Pi 5 the Hantro G2 + CMA allocation for 4K60 HEVC
is still mid-init at 66 ms, the stop() interrupts it, and the
QMediaPlayer leaves position pegged at 0 for the entire 60 s
asset_loop window — black screen, 3600 dropped frames. Subsequent
plays of the same clip work because partial decoder state
persists.
Skip hideVideoSurface when preUri == 'null'. Real image URIs
still tear down the video.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



Issues Fixed
Closes #2904. Stacked on top of #2885 — base
perf/pi-vo-gpu-drm. Merge order: 2885 first.Description
The viewer's Qt6 video path was two processes contending for one framebuffer:
AnthiasWebview(Qt) holding the surface, and an externalmpvsubprocess painting video via--vo=drm(linuxfb) or--vo=gpu --gpu-context=wayland(cage). PR #2885 documented 600-2800 vo drops per 60 s clip on Pi 4 even with HW decode engaged.This PR moves video playback inside AnthiasWebview's Qt process via QtMultimedia (
QMediaPlayer+QVideoWidget). Single Qt process, single GL/Wayland surface, no more mpv subprocess.Implementation arc
I tried two embedding approaches; the second is what's shipping:
libmpv via
mpv_render_context— engaged HW decode correctly on every board (verbose mpv logs confirmdrm-copyengaged on rpi-hevc-dec via/dev/video19,v4l2m2m-copyon bcm2835-codec via/dev/video10) but Pi 4 frame drops stayed at 600-2800/60 s. Root cause: libmpv-render uploads decoded frames to a GL texture, thenQOpenGLWidgetblits that into an offscreen FBO, then Qt's compositor blits the FBO into the eglfs window — three GPU passes per frame on a V3D 6.0 that can't sustain that at 60 fps. AQOpenGLWindowworkaround to bypass the FBO crashed because eglfs is single-native-window-per-process. Branch history preserves the experiment; the final tree replaced it.QtMultimedia
QMediaPlayer+QVideoWidget— ships. QVideoWidget paints inside MainWindow's existing eglfs native window (no second QWindow), Qt 6.8's ffmpeg-backed multimedia pipeline routes decoded frames throughQVideoSinkwithout the FBO indirection. The +rpt1 ffmpeg (already pinned in feat(viewer,server): per-board HW decode dispatch + codec gate on upload #2885's_rpt1-ffmpeg-pin.j2) carries--enable-v4l2-request --enable-v4l2-m2m, so the existing Pi-family hardware decoders (rpi-hevc-dec, bcm2835-codec, Hantro G2 on Pi 5, rkvdec on Rock Pi 4) all engage automatically via libavcodec without any per-codec dispatch from the application.Real-device validation (BBB pack)
All three reachable Qt6 boards measured against the same 8-clip BBB pack (1080p / 4K, 30 / 60 fps, in H.264 and HEVC, each clip ~60 s @ ~8 Mbps). Drop counts come from the in-process counter in
VideoView::onVideoFrameDelivered()(frames the QVideoSink received) compared tocontainer_fps × position_s, written to/data/.anthias/mpv-stats.logper play (filename held for compat with the test-bed grep workflow). The numbers below are the median across n full-clip plays in a ≥1200 s observation window (Pi 5 ran 5 cycles; Pi 4 / x86 ran 2-3 cycles given the 8-clip rotation).Format:
dropped / expected frames (n = full-clip plays).*= derived from the viewer's STOP log line whenasset_looppreempted the clip at its declared duration beforeQMediaPlayer::EndOfMediacould fire (expected = position_ms × container_fps).The cells doing the heavy lifting for #2904:
v4l2m2m-copydecoder is at real-time, and removing the QOpenGLWidget FBO indirection let V3D keep up with the present pipeline.rpi-hevc-dec(/dev/video19) drives it viav4l2_requestfrom the +rpt1libavcodec, and QtMultimedia's ffmpeg backend routes the decoded surfaces straight toQVideoSink.Pi 4's 4K H.264 rows (124 and 242 drops/min) are the only outliers. Stream position at the 60 s wall-clock mark only reached 48.8 s (4K30) / 28.5 s (4K60) — that's the bcm2835-codec H.264 path running ≈ 80 % / ≈ 47 % of real time, not a presentation-side drop. This is a hardware limitation that pre-existed PR #2885;
_HW_DECODE_VIDEO_CODECS['pi4-64']would still accept these uploads, but the rest of the matrix shows the gate's intent (H.264 ≤ 1080p, HEVC at any resolution) is already what the hardware can sustain. Out of scope for #2904.Rock Pi 4 validation is deferred — SSH banner exchange has been timing out on the testbed since the second 4K HEVC ingest stalled there earlier today. The arm64 image (
69bb873-arm64) carries the sameVideoViewcode path the other three boards just validated. Tracking the re-test separately.Architecture changes
src/anthias_webview/—VideoViewis now aQWidgethostingQMediaPlayer+QVideoWidget+QAudioOutput. MainWindow exposes the same D-Bus surface (playVideo/stopVideo/videoEnded) as the libmpv variant, so the Python contract is untouched.src/anthias_viewer/media_player.py— option dict shrinks from {hwdec,video-sync,vd-lavc-threads,audio-device,video-rotate} to just {audio-device,video-rotate}._PI_HWDEC_BY_CODEC,_pi_hwdec_for_uri, and_probe_video_codec(ffprobe subprocess) are gone — libavcodec's auto-selection covers the per-board decoder dispatch.docker/Dockerfile.viewer.j2— dropslibmpv2, addslibqt6multimedia6,libqt6multimediawidgets6,qt6-multimedia-dev. Pi 4 switchesQT_QPA_PLATFORMfromlinuxfbtoeglfs(libmpv-render needed a GL context; QtMultimedia keeps benefitting from one). The eglfs KMS config pinning 1920x1080 is shipped atdocker/eglfs-kms-pi4.json.tests/test_media_player.py— argv assertions become D-Bus options-dict assertions; per-codec dispatch tests dropped; rotation / audio / proxy / VLC-fallback tests preserved. 63 cases green.src/anthias_webview/tests/(new) — QtTest unit suite forVideoView(constructor builds player, stop idempotent, play with empty/unknown audio device, video-rotate passthrough).bin/test_webview_cpp.shbuilds + runs them underQT_QPA_PLATFORM=offscreen. CI integration is a follow-up.Key fixes during validation
view_image('null')was stopping the freshly-started video.src/anthias_viewer/__init__.py:495callsview_image('null')AFTERmedia_player.play()to clear any leftover image/web background. My initialView::loadImagealways calledhideVideoSurface(), which stopped the QMediaPlayer 66 ms after PLAYING fired. On Pi 4 this was invisible (decoder past init); on Pi 5 it killed the Hantro G2 mid-CMA-allocation for 4K60 HEVC — 0 frames in 60 s. Fix: skiphideVideoSurfacefor the'null'sentinel (commit).QUrl(uri)with scheme-less local paths refused to load.QMediaPlayersilently rejected/data/anthias_assets/abc.mp4because the QUrl had no scheme. UseQUrl::fromLocalFilefor any path starting with/.GLib.Variantwrap for D-Busa{sv}— pydbus refuses to auto-coercestrtoGLib.Variantwhen the slot is declared withQVariantMap. Wrap each option value via_marshal_dbus_options(regression test guards it).Notes for upgrade
Pre-PR-#2885 testbed images carry
QT_QPA_PLATFORM=linuxfbhardcoded in the device's deployeddocker-compose.yml. The currentdocker-compose.yml.tmplno longer pins the platform —bin/upgrade_containers.shregenerates compose from the template, so devices on the new image automatically pick up the Dockerfile's per-board default.Known pre-existing issue (not introduced here)
view_videoinsrc/anthias_viewer/__init__.py:497-505has a race:skip_event.clear() → wait(timeout=duration). If something else (e.g. an API state change firingreload) setsskip_eventbetween theclear()andwait(), the very first asset after a viewer restart can be stopped immediately. Not introduced by this PR; worth filing separately.Checklist